<html>
<head><meta charset="utf-8"><title>DAG model vs header inclusion model · t-compiler/wg-pipelining · Zulip Chat Archive</title></head>
<h2>Stream: <a href="https://rust-lang.github.io/zulip_archive/stream/195180-t-compiler/wg-pipelining/index.html">t-compiler/wg-pipelining</a></h2>
<h3>Topic: <a href="https://rust-lang.github.io/zulip_archive/stream/195180-t-compiler/wg-pipelining/topic/DAG.20model.20vs.20header.20inclusion.20model.html">DAG model vs header inclusion model</a></h3>

<hr>

<base href="https://rust-lang.zulipchat.com">

<head><link href="https://rust-lang.github.io/zulip_archive/style.css" rel="stylesheet"></head>

<a name="164355909"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/195180-t-compiler/wg-pipelining/topic/DAG%20model%20vs%20header%20inclusion%20model/near/164355909" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Vadim Petrochenkov <a href="https://rust-lang.github.io/zulip_archive/stream/195180-t-compiler/wg-pipelining/topic/DAG.20model.20vs.20header.20inclusion.20model.html#164355909">(Apr 27 2019 at 22:10)</a>:</h4>
<p>A fun experimental paper from the recent C++ committee mailing - "Are modules fast?" (<a href="http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2019/p1441r1.pdf" target="_blank" title="http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2019/p1441r1.pdf">http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2019/p1441r1.pdf</a>)</p>



<a name="164355981"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/195180-t-compiler/wg-pipelining/topic/DAG%20model%20vs%20header%20inclusion%20model/near/164355981" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Vadim Petrochenkov <a href="https://rust-lang.github.io/zulip_archive/stream/195180-t-compiler/wg-pipelining/topic/DAG.20model.20vs.20header.20inclusion.20model.html#164355981">(Apr 27 2019 at 22:12)</a>:</h4>
<p>The proposed C++ modules are very similar to our packages and they similarly don't scale to massively parallel builds.</p>



<a name="164356076"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/195180-t-compiler/wg-pipelining/topic/DAG%20model%20vs%20header%20inclusion%20model/near/164356076" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Vadim Petrochenkov <a href="https://rust-lang.github.io/zulip_archive/stream/195180-t-compiler/wg-pipelining/topic/DAG.20model.20vs.20header.20inclusion.20model.html#164356076">(Apr 27 2019 at 22:15)</a>:</h4>
<p>I wonder if Cargo eventually ends up compiling leaf crates multiple times given enough parallelism to compensate deep dependency trees, that would be kind of a reasonable return to the header inclusion model.</p>



<a name="164356546"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/195180-t-compiler/wg-pipelining/topic/DAG%20model%20vs%20header%20inclusion%20model/near/164356546" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Vadim Petrochenkov <a href="https://rust-lang.github.io/zulip_archive/stream/195180-t-compiler/wg-pipelining/topic/DAG.20model.20vs.20header.20inclusion.20model.html#164356546">(Apr 27 2019 at 22:28)</a>:</h4>
<p>Or maybe that's a nonsense, unless builds span multiple machines.</p>



<a name="164460025"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/195180-t-compiler/wg-pipelining/topic/DAG%20model%20vs%20header%20inclusion%20model/near/164460025" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> mw <a href="https://rust-lang.github.io/zulip_archive/stream/195180-t-compiler/wg-pipelining/topic/DAG.20model.20vs.20header.20inclusion.20model.html#164460025">(Apr 29 2019 at 15:06)</a>:</h4>
<p>interesting</p>



<a name="164565294"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/195180-t-compiler/wg-pipelining/topic/DAG%20model%20vs%20header%20inclusion%20model/near/164565294" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Alex Crichton <a href="https://rust-lang.github.io/zulip_archive/stream/195180-t-compiler/wg-pipelining/topic/DAG.20model.20vs.20header.20inclusion.20model.html#164565294">(Apr 30 2019 at 18:52)</a>:</h4>
<p>FWIW I've often though that Cargo's model of compilation is basically unsuitable for the "we have infinite parallelism" compilation model (aka distributed compilation)</p>



<a name="164565351"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/195180-t-compiler/wg-pipelining/topic/DAG%20model%20vs%20header%20inclusion%20model/near/164565351" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Alex Crichton <a href="https://rust-lang.github.io/zulip_archive/stream/195180-t-compiler/wg-pipelining/topic/DAG.20model.20vs.20header.20inclusion.20model.html#164565351">(Apr 30 2019 at 18:53)</a>:</h4>
<p>My gut is that we would have to rethink quite a lot to really take advantage of infinite parallelism, and all of cargo/rustc/parallelism we have today is really heavily optimized for "reasonable parallelism on one machine"</p>



<a name="164565371"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/195180-t-compiler/wg-pipelining/topic/DAG%20model%20vs%20header%20inclusion%20model/near/164565371" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Alex Crichton <a href="https://rust-lang.github.io/zulip_archive/stream/195180-t-compiler/wg-pipelining/topic/DAG.20model.20vs.20header.20inclusion.20model.html#164565371">(Apr 30 2019 at 18:53)</a>:</h4>
<p>although that is also starting to show wear and tear with lots of cpus and big crate graphs...</p>



<a name="164582722"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/195180-t-compiler/wg-pipelining/topic/DAG%20model%20vs%20header%20inclusion%20model/near/164582722" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> njn <a href="https://rust-lang.github.io/zulip_archive/stream/195180-t-compiler/wg-pipelining/topic/DAG.20model.20vs.20header.20inclusion.20model.html#164582722">(Apr 30 2019 at 22:34)</a>:</h4>
<p>I have a machine with 14 physical cores and the difference between C++ compilation and Rust compilation is huge (e.g. in Firefox). Compilation of Rust code doesn't get close to saturating the cores.</p>



<a name="164583512"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/195180-t-compiler/wg-pipelining/topic/DAG%20model%20vs%20header%20inclusion%20model/near/164583512" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> njn <a href="https://rust-lang.github.io/zulip_archive/stream/195180-t-compiler/wg-pipelining/topic/DAG.20model.20vs.20header.20inclusion.20model.html#164583512">(Apr 30 2019 at 22:45)</a>:</h4>
<p>That's my primary motivation for working on pipelining.</p>



<a name="164622405"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/195180-t-compiler/wg-pipelining/topic/DAG%20model%20vs%20header%20inclusion%20model/near/164622405" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> eddyb <a href="https://rust-lang.github.io/zulip_archive/stream/195180-t-compiler/wg-pipelining/topic/DAG.20model.20vs.20header.20inclusion.20model.html#164622405">(May 01 2019 at 13:19)</a>:</h4>
<p>IMO fixing <a href="https://github.com/rust-lang/rust/issues/54712#issuecomment-485656192" target="_blank" title="https://github.com/rust-lang/rust/issues/54712#issuecomment-485656192">https://github.com/rust-lang/rust/issues/54712#issuecomment-485656192</a> and <a href="https://github.com/rust-lang/rust/issues/58485#issuecomment-485658943" target="_blank" title="https://github.com/rust-lang/rust/issues/58485#issuecomment-485658943">https://github.com/rust-lang/rust/issues/58485#issuecomment-485658943</a> would provide a better improvement than pipelining</p>



<a name="164622434"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/195180-t-compiler/wg-pipelining/topic/DAG%20model%20vs%20header%20inclusion%20model/near/164622434" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> eddyb <a href="https://rust-lang.github.io/zulip_archive/stream/195180-t-compiler/wg-pipelining/topic/DAG.20model.20vs.20header.20inclusion.20model.html#164622434">(May 01 2019 at 13:19)</a>:</h4>
<p>Rust also saturates all the cores during codegen, and ofc during running tests</p>



<a name="164622500"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/195180-t-compiler/wg-pipelining/topic/DAG%20model%20vs%20header%20inclusion%20model/near/164622500" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> eddyb <a href="https://rust-lang.github.io/zulip_archive/stream/195180-t-compiler/wg-pipelining/topic/DAG.20model.20vs.20header.20inclusion.20model.html#164622500">(May 01 2019 at 13:20)</a>:</h4>
<p>but this is with incremental. for batch compilation, pipelining should be a clear win</p>



<a name="164622533"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/195180-t-compiler/wg-pipelining/topic/DAG%20model%20vs%20header%20inclusion%20model/near/164622533" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> eddyb <a href="https://rust-lang.github.io/zulip_archive/stream/195180-t-compiler/wg-pipelining/topic/DAG.20model.20vs.20header.20inclusion.20model.html#164622533">(May 01 2019 at 13:21)</a>:</h4>
<p>(I assumed you meant "compiling rustc" but that's not all the Rust code out there, heh)</p>



<a name="164631390"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/195180-t-compiler/wg-pipelining/topic/DAG%20model%20vs%20header%20inclusion%20model/near/164631390" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Alex Crichton <a href="https://rust-lang.github.io/zulip_archive/stream/195180-t-compiler/wg-pipelining/topic/DAG.20model.20vs.20header.20inclusion.20model.html#164631390">(May 01 2019 at 15:22)</a>:</h4>
<p><span class="user-mention" data-user-id="119009">@eddyb</span> your improvements are rustc-specific though I think, right? They're not generally applicable to crate graphs in the large?</p>



<a name="164631431"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/195180-t-compiler/wg-pipelining/topic/DAG%20model%20vs%20header%20inclusion%20model/near/164631431" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> eddyb <a href="https://rust-lang.github.io/zulip_archive/stream/195180-t-compiler/wg-pipelining/topic/DAG.20model.20vs.20header.20inclusion.20model.html#164631431">(May 01 2019 at 15:23)</a>:</h4>
<p>that last message from me should've started with "oops"</p>



<a name="164631561"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/195180-t-compiler/wg-pipelining/topic/DAG%20model%20vs%20header%20inclusion%20model/near/164631561" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> eddyb <a href="https://rust-lang.github.io/zulip_archive/stream/195180-t-compiler/wg-pipelining/topic/DAG.20model.20vs.20header.20inclusion.20model.html#164631561">(May 01 2019 at 15:24)</a>:</h4>
<p>(I had misread what <span class="user-mention" data-user-id="120989">@nnethercote</span> had said. and now I can't edit my messages to make it even clearer)</p>



<hr><p>Last updated: Aug 07 2021 at 22:04 UTC</p>
</html>